{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 计算电影相似度，将电影信息映射成文本向量，计算向量间的余弦相似度"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "' \\n@author: 梅楚鹤\\n@contact: 1532744860@qq.com\\n@software: jupyter\\n@time: 19-5-10 9:00am\\n'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#!/usr/bin/python  \n",
    "# -*- coding:utf-8 _*-\n",
    "\"\"\" \n",
    "@author: 梅楚鹤\n",
    "@contact: 1532744860@qq.com\n",
    "@software: jupyter\n",
    "@time: 19-5-10 9:00am\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.ml.feature import Word2Vec\n",
    "from pyspark import HiveContext, SQLContext, SparkContext, SparkConf\n",
    "from pyspark.sql import SparkSession\n",
    "import pyspark"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "sc = SparkContext(appName=\"pyspark mysql demo\")\n",
    "sqlContext = SQLContext(sc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-----+--------------+------------------+-----------------+-------------------------+-----------------------------------+-------------------------+--------------------+--------+----------+-------------------------------------+-------------------+-----+\n",
      "|   id|        ztitle|      release_time|       mv_country|              mv_director|                            mv_star|          mv_scriptwriter|            mv_image|mv_score|mv_numbers|                      mv_introduction|            mv_type|intro|\n",
      "+-----+--------------+------------------+-----------------+-------------------------+-----------------------------------+-------------------------+--------------------+--------+----------+-------------------------------------+-------------------+-----+\n",
      "|   49|      大闹天宫|2017-10-13大陆上映|         中国大陆|                     速达|                   李扬\\陈道明\\姚晨|                     速达|https://p1.meitua...|     8.3|      1362|在遥远的东方傲来国，一枚承受了日精...|               动画|     |\n",
      "|  123|          龙猫|2018-12-14大陆上映|             日本|                   宫崎骏|             秦岚\\糸井重里\\岛本须美|                   宫崎骏|https://p0.meitua...|     9.1|    11.2万| 小月（日高法子 配音）的母亲（岛本...|动画,冒险,奇幻,家庭|     |\n",
      "|  784|        甜蜜蜜|2015-02-13大陆上映|         中国香港|                   陈可辛|                 黎明\\张曼玉\\曾志伟|                   陈可辛|https://p1.meitua...|     9.2|     12562| 改革开放初期，黎小军（黎明 饰）告...|          剧情,爱情|     |\n",
      "|  797|兔子镇的火狐狸|2015-10-30大陆上映|         中国大陆|                   葛水英|                 乔诗语\\宣晓鸣\\陈浩|                   葛水英|https://p0.meitua...|     8.2|      2408|兔子镇和狐狸镇是几百年的世代宿敌。...|     动画,冒险,奇幻|     |\n",
      "|  905|    一代宗师3D|2015-01-08大陆上映|中国大陆,中国香港|                   王家卫|                 梁朝伟\\章子怡\\张震|                   王家卫|https://p0.meitua...|     8.2|     39277| 广东佛山人叶问（梁朝伟 饰），年少...|     剧情,动作,传记|     |\n",
      "| 1200|        功夫3D|2015-01-15大陆上映|中国大陆,中国香港|                   周星驰|                 周星驰\\黄圣依\\元华|                   周星驰|https://p0.meitua...|     9.3|     26758|    1940年代的上海，自小受尽欺辱的...|     喜剧,动作,犯罪|     |\n",
      "| 1247|          教父|2015-04-18大陆上映|             美国|     弗朗西斯·福特·科波拉|马龙·白兰度\\阿尔·帕西诺\\詹姆斯·肯恩|     弗朗西斯·福特·科波拉|https://p0.meitua...|     9.3|      1193|      40年代的美国，“教父”维托·唐·...|     犯罪,剧情,惊悚|     |\n",
      "| 1287|爱在日落黄昏时|2015-04-20大陆上映|             美国|          理查德·林克莱特|伊桑·霍克\\朱莉·德尔佩\\弗农·多布切夫|          理查德·林克莱特|https://p1.meitua...|     8.7|       195|影片讲述了九年过后一段缘分的再续。...|          剧情,爱情|     |\n",
      "| 1332|      美国往事|2015-04-23大陆上映|      意大利,美国|            赛尔乔·莱翁内|  罗伯特·德尼罗\\詹姆斯·伍兹\\伊丽...|            赛尔乔·莱翁内|https://p1.meitua...|     9.1|       363|  二十年代的美国，纽约少年“面条”（...|          犯罪,剧情|     |\n",
      "| 2972|         教父3|2015-04-20大陆上映|             美国|     弗朗西斯·福特·科波拉|  阿尔·帕西诺\\安迪·加西亚\\索菲亚...|     弗朗西斯·福特·科波拉|https://p0.meitua...|     8.6|       224|    转眼间已经是1979年，第二代教父...|     犯罪,剧情,惊悚|     |\n",
      "| 4587|爱在黎明破晓前|2015-04-20大陆上映| 美国,奥地利,瑞士|          理查德·林克莱特|  伊桑·霍克\\朱莉·德尔佩\\安德莉亚...|          理查德·林克莱特|https://p1.meitua...|     8.7|       285|影片讲述了一对陌生男女一天之内在异...|          剧情,爱情|     |\n",
      "| 4710|      灵性之光|2015-04-18大陆上映|             波兰|        克日什托夫·扎努西|             Jan Skotnicki\\马辛·...|        克日什托夫·扎努西|https://p1.meitua...|     7.8|         1|赞努西打开国际影坛大门的早期代表作...|               剧情|     |\n",
      "| 7641|      机票情缘|      2016大陆上映|             美国|                  唐·罗斯|  本·阿弗莱克\\格温妮斯·帕特洛\\娜...|                  唐·罗斯|https://p0.meitua...|     6.2|         3| 一切本来都完美无瑕，巴迪（本·阿弗...|          剧情,爱情|     |\n",
      "| 9839|    石榴的颜色|2015-06-14大陆上映|             苏联|        谢尔盖·帕拉杰诺夫|       索菲柯·齐阿乌列里\\Giorgi ...|        谢尔盖·帕拉杰诺夫|https://p1.meitua...|     8.2|         1|   影片部分根据18世纪亚美尼亚诗人S...|     传记,剧情,音乐|     |\n",
      "| 9927|      杀人短片|2015-04-20大陆上映|             波兰|克日什托夫·基耶斯洛夫斯基| 米罗斯洛·巴卡\\克日什托夫·格洛比...|克日什托夫·基耶斯洛夫斯基|https://p0.meitua...|     8.3|        15|            Jacek Lazar（米罗斯洛·...|          犯罪,剧情|     |\n",
      "|10757|          缘份|2016-03-25大陆上映|         中国香港|                   黄泰来|               张国荣\\张曼玉\\梅艳芳|                   黄泰来|https://p1.meitua...|     8.6|     28862|在新开通的地下铁中，青年陈保罗（张...|     剧情,喜剧,爱情|     |\n",
      "|11237|      阿飞正传|2018-06-25大陆上映|         中国香港|                   王家卫|               张国荣\\张曼玉\\刘德华|                   王家卫|https://p0.meitua...|     8.8|     21966|英俊不凡的旭仔阿飞是上海移民，他从...|     剧情,爱情,犯罪|     |\n",
      "|11792|          侠女|2017-11-03大陆上映|中国台湾,中国香港|                   胡金铨|                     徐枫\\石隽\\白鹰|                   胡金铨|https://p1.meitua...|     8.1|        49| 没落书生顾省斋（石隽 饰）在小镇经...|剧情,动作,武侠,古装|     |\n",
      "|11904|        铁道员|2015-06-14大陆上映|             日本|                 降旗康男|             高仓健\\大竹忍\\广末凉子|                 降旗康男|https://p1.meitua...|     8.1|        13|幌舞作为煤矿产地曾繁华过，无奈现今...|          剧情,家庭|     |\n",
      "|13457|        龙凤斗|2018-02-10大陆上映|         中国香港|                   杜琪峰|               刘德华\\郑秀文\\吴嘉龙|                   杜琪峰|https://p1.meitua...|     7.9|      1301|  盗生（刘德华 饰）与盗太（郑秀文 ...|     剧情,喜剧,爱情|     |\n",
      "+-----+--------------+------------------+-----------------+-------------------------+-----------------------------------+-------------------------+--------------------+--------+----------+-------------------------------------+-------------------+-----+\n",
      "only showing top 20 rows\n",
      "\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "# 创建连接获取数据\n",
    "df_movie=sqlContext.read.format(\"jdbc\").option(\"url\", \"jdbc:mysql://localhost:3308/meich_db\")\\\n",
    ".option(\"dbtable\", \"tq2\").option(\"user\", \"root\").option(\"password\", \"Mysql_08\").load()\n",
    " \n",
    "# 输出数据\n",
    "# print (\"dataframe_mysql.collect()\",dataframe_mysql.collect())\n",
    "# dataframe_mysql.registerTempTable(\"temp_table\")\n",
    "print(df_movie.show())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Row(id=49, ztitle='大闹天宫', release_time='2017-10-13大陆上映', mv_country='中国大陆', mv_director='速达', mv_star='李扬\\\\陈道明\\\\姚晨', mv_scriptwriter='速达', mv_image='https://p1.meituan.net/movie/0e2400f144c69fe28edb36a8ec91784a744391.jpg@464w_644h_1e_1c', mv_score='8.3', mv_numbers='1362', mv_introduction='在遥远的东方傲来国，一枚承受了日精月华的石头突然炸裂，从里面蹦出了一只石猴（李扬 饰）。石猴在附近的花果山占山为王，逍遥自在，好不快活。只可惜他虽天赋神力，却没有一件得心应手的兵器。在老猴的建议下，美猴王入龙宫，强从东海龙王（陈凯歌 饰）那里讨来定海神针金箍棒。龙王一纸诉状告到天庭，玉帝（陈道明 饰）接受太白金星（张国立 饰）的建议，先后将美猴王骗到天宫作弼马温、管理蟠桃园。发觉被骗的美猴王怒砸马栏、破坏蟠桃宴，返回花果山自命齐天大圣。由此和十万天兵天将展开了惊心动魄的大战……', mv_type='动画', intro='')]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_movie.head(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 提取片名、导演、主演、时间、标签等组成特征向量"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "def onehotencode(df, s1, s2, temp):\n",
    "    from pyspark.ml.feature import OneHotEncoder, StringIndexer\n",
    "    stringIndexer = StringIndexer(inputCol=s1, outputCol=temp)\n",
    "    model = stringIndexer.fit(df)\n",
    "    indexed = model.transform(df)\n",
    "    encoder = OneHotEncoder(inputCol=temp, outputCol=s2)\n",
    "    encoded = encoder.transform(indexed)\n",
    "    encoded.select(s2).show()\n",
    "    return encoded"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------------------+\n",
      "|           direcVec|\n",
      "+-------------------+\n",
      "|  (2012,[73],[1.0])|\n",
      "| (2012,[474],[1.0])|\n",
      "| (2012,[636],[1.0])|\n",
      "| (2012,[849],[1.0])|\n",
      "| (2012,[263],[1.0])|\n",
      "|  (2012,[56],[1.0])|\n",
      "| (2012,[242],[1.0])|\n",
      "|  (2012,[22],[1.0])|\n",
      "|(2012,[1377],[1.0])|\n",
      "| (2012,[242],[1.0])|\n",
      "|  (2012,[22],[1.0])|\n",
      "| (2012,[160],[1.0])|\n",
      "| (2012,[545],[1.0])|\n",
      "| (2012,[342],[1.0])|\n",
      "|(2012,[1809],[1.0])|\n",
      "|(2012,[1485],[1.0])|\n",
      "| (2012,[263],[1.0])|\n",
      "|  (2012,[89],[1.0])|\n",
      "| (2012,[451],[1.0])|\n",
      "| (2012,[265],[1.0])|\n",
      "+-------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df_dir = onehotencode(df_movie, \"mv_director\", \"direcVec\", \"direcindex\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-----------------+\n",
      "|          counVec|\n",
      "+-----------------+\n",
      "|  (334,[0],[1.0])|\n",
      "|  (334,[2],[1.0])|\n",
      "|  (334,[8],[1.0])|\n",
      "|  (334,[0],[1.0])|\n",
      "|  (334,[4],[1.0])|\n",
      "|  (334,[4],[1.0])|\n",
      "|  (334,[1],[1.0])|\n",
      "|  (334,[1],[1.0])|\n",
      "| (334,[80],[1.0])|\n",
      "|  (334,[1],[1.0])|\n",
      "|(334,[118],[1.0])|\n",
      "| (334,[33],[1.0])|\n",
      "|  (334,[1],[1.0])|\n",
      "|(334,[310],[1.0])|\n",
      "| (334,[33],[1.0])|\n",
      "|  (334,[8],[1.0])|\n",
      "|  (334,[8],[1.0])|\n",
      "| (334,[69],[1.0])|\n",
      "|  (334,[2],[1.0])|\n",
      "|  (334,[8],[1.0])|\n",
      "+-----------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "dir_cou = onehotencode(df_dir, \"mv_country\", \"counVec\", \"counIndex\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------------------+\n",
      "|           direcVec|\n",
      "+-------------------+\n",
      "|  (2012,[73],[1.0])|\n",
      "| (2012,[474],[1.0])|\n",
      "| (2012,[636],[1.0])|\n",
      "| (2012,[849],[1.0])|\n",
      "| (2012,[263],[1.0])|\n",
      "|  (2012,[56],[1.0])|\n",
      "| (2012,[242],[1.0])|\n",
      "|  (2012,[22],[1.0])|\n",
      "|(2012,[1377],[1.0])|\n",
      "| (2012,[242],[1.0])|\n",
      "|  (2012,[22],[1.0])|\n",
      "| (2012,[160],[1.0])|\n",
      "| (2012,[545],[1.0])|\n",
      "| (2012,[342],[1.0])|\n",
      "|(2012,[1809],[1.0])|\n",
      "|(2012,[1485],[1.0])|\n",
      "| (2012,[263],[1.0])|\n",
      "|  (2012,[89],[1.0])|\n",
      "| (2012,[451],[1.0])|\n",
      "| (2012,[265],[1.0])|\n",
      "+-------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from pyspark.ml.feature import OneHotEncoder, StringIndexer\n",
    " \n",
    "stringIndexer = StringIndexer(inputCol=\"mv_director\", outputCol=\"director_Index\")\n",
    "model = stringIndexer.fit(df_movie)\n",
    "indexed = model.transform(df_movie)\n",
    "\n",
    "encoder = OneHotEncoder(inputCol=\"director_Index\", outputCol=\"direcVec\")\n",
    "encoded = encoder.transform(indexed)\n",
    "encoded.select('direcVec').show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.mllib.feature import Word2Vec\n",
    "from pyspark.ml.feature import Word2Vec\n",
    "from pyspark.ml.feature import CountVectorizer, CountVectorizerModel, Tokenizer, RegexTokenizer, StopWordsRemover"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "import jieba\n",
    "from pyspark.sql.functions import UserDefinedFunction\n",
    "from pyspark.sql.types import StringType\n",
    "\n",
    "def seg(s):\n",
    "    fenci_text = jieba.cut(s)\n",
    "    result = \" \".join(fenci_text)\n",
    "    return result\n",
    "\n",
    "# 根据python的返回值类型定义好spark对应的数据类型\n",
    "# python函数中返回的是string，对应的pyspark是StringType\n",
    "segUDF=UserDefinedFunction(seg, StringType()) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 使用自定义函数\n",
    "df_seg = df_movie.withColumn('introduction',segUDF('mv_introduction'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "StructType(List(StructField(id,IntegerType,true),StructField(ztitle,StringType,true),StructField(release_time,StringType,true),StructField(mv_country,StringType,true),StructField(mv_director,StringType,true),StructField(mv_star,StringType,true),StructField(mv_scriptwriter,StringType,true),StructField(mv_image,StringType,true),StructField(mv_score,StringType,true),StructField(mv_numbers,StringType,true),StructField(mv_introduction,StringType,true),StructField(mv_type,StringType,true),StructField(intro,StringType,true),StructField(introduction,StringType,true)))"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_seg.schema"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### word2vec"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "def word2vec(df, inputcol, outputcol):\n",
    "    from pyspark.mllib.feature import Word2Vec\n",
    "    from pyspark.ml.feature import Word2Vec\n",
    "    from pyspark.ml.feature import CountVectorizer, CountVectorizerModel, Tokenizer, RegexTokenizer, StopWordsRemover\n",
    "    # 使用自定义函数\n",
    "    df_seg = df.withColumn(inputcol,segUDF(inputcol))\n",
    "    df_seg.drop('words')\n",
    "    tokenizer = Tokenizer(inputCol=inputcol, outputCol='words')\n",
    "    t_words = tokenizer.transform(df_seg)\n",
    "    t_words.select('words').head()\n",
    "    #4.将文本向量转换成稀疏表示的数值向量（字符频率向量）\n",
    "    cv = CountVectorizer(inputCol=\"words\", outputCol=\"features\", vocabSize=5, minDF=2.0)\n",
    "    t_words.drop(\"features\")\n",
    "    cv_model = cv.fit(t_words)\n",
    "    cv_result = cv_model.transform(t_words)\n",
    "    #5.将tokenizer得到的分词结果转换数字向量\n",
    "    word2Vec = Word2Vec(vectorSize=100, minCount=0, inputCol=\"words\", outputCol=outputcol)\n",
    "    w2v_model = word2Vec.fit(cv_result)\n",
    "    result = w2v_model.transform(cv_result)\n",
    "    for feature in result.select(outputcol).take(3):\n",
    "        print(feature)\n",
    "        return t_words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Row(result=DenseVector([-0.0368, 0.0521, 0.0304, -0.0058, 0.0276, -0.0319, -0.0348, -0.0128, 0.0229, 0.0059, -0.0312, -0.027, 0.0072, 0.0115, 0.0291, -0.0271, -0.0305, 0.0622, 0.0813, 0.0124, 0.0212, -0.0282, -0.0005, -0.0046, 0.0424, 0.0021, 0.0344, 0.011, 0.0171, -0.0007, 0.0123, -0.0303, 0.0058, -0.0522, -0.0117, 0.0237, 0.0354, -0.0375, -0.0139, 0.0017, 0.0213, -0.0018, -0.0293, -0.02, -0.0264, -0.0098, -0.0628, 0.0543, -0.015, 0.0543, -0.0379, 0.0063, -0.0319, 0.0276, -0.0604, 0.0106, 0.0399, -0.0273, 0.0378, 0.0363, -0.0474, 0.002, -0.041, -0.01, -0.0365, -0.0443, 0.0374, 0.0136, 0.0234, 0.0143, -0.0311, 0.0007, -0.0522, 0.0167, -0.0048, -0.0202, -0.0521, 0.0203, 0.0099, -0.0341, 0.0294, 0.0025, -0.0491, -0.0101, 0.0351, -0.0634, 0.0101, -0.0118, 0.0365, 0.0212, -0.0119, -0.0628, -0.0378, -0.0238, 0.0514, -0.0297, -0.0109, 0.0135, 0.0383, 0.0447]))\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "DataFrame[id: int, ztitle: string, release_time: string, mv_country: string, mv_director: string, mv_star: string, mv_scriptwriter: string, mv_image: string, mv_score: string, mv_numbers: string, mv_introduction: string, mv_type: string, intro: string, words: array<string>]"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "word2vec(df_movie, \"mv_introduction\", \"result\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Row(words=['在', '遥远', '的', '东方', '傲来国', '，', '一枚', '承受', '了', '日', '精', '月华', '的', '石头', '突然', '炸裂', '，', '从', '里面', '蹦出', '了', '一只', '石猴', '（', '李扬', '', '', '饰', '）', '。', '石猴', '在', '附近', '的', '花果山', '占山为王', '，', '逍遥自在', '，', '好', '不', '快活', '。', '只', '可惜', '他', '虽', '天赋', '神力', '，', '却', '没有', '一件', '得心应手', '的', '兵器', '。', '在', '老猴', '的', '建议', '下', '，', '美猴王', '入', '龙宫', '，', '强', '从', '东海龙王', '（', '陈凯歌', '', '', '饰', '）', '那里', '讨来', '定海神针', '金箍棒', '。', '龙王', '一纸', '诉状', '告到', '天庭', '，', '玉帝', '（', '陈道明', '', '', '饰', '）', '接受', '太白金星', '（', '张国立', '', '', '饰', '）', '的', '建议', '，', '先后', '将', '美猴王', '骗', '到', '天宫', '作弼', '马温', '、', '管理', '蟠桃', '园', '。', '发觉', '被', '骗', '的', '美猴王', '怒', '砸', '马栏', '、', '破坏', '蟠桃', '宴', '，', '返回', '花果山', '自命', '齐天大圣', '。', '由此', '和', '十万', '天兵天将', '展开', '了', '惊心动魄', '的', '大战', '…', '…'])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#3.使用tokenizer分词\n",
    "tokenizer = Tokenizer(inputCol=\"introduction\", outputCol=\"words\")\n",
    "t_words = tokenizer.transform(df_seg)\n",
    "t_words.select('words').head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "#4.将文本向量转换成稀疏表示的数值向量（字符频率向量）\n",
    "cv = CountVectorizer(inputCol=\"words\", outputCol=\"features\", vocabSize=5, minDF=2.0)\n",
    "cv_model = cv.fit(t_words)\n",
    "cv_result = cv_model.transform(t_words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Row(result=DenseVector([-0.004, -0.0329, 0.0336, -0.0622, 0.0102, -0.0707, 0.0104, -0.0022, 0.047, -0.0213, -0.041, 0.0436, 0.0708, 0.0098, 0.0238, 0.0504, -0.0099, 0.0084, -0.0105, 0.0438, 0.0151, -0.0114, 0.0332, -0.011, 0.0298, 0.0078, 0.0365, -0.0076, 0.0023, -0.0035, -0.0121, -0.0259, 0.0458, -0.017, -0.0015, -0.0189, 0.0158, 0.0286, 0.0211, 0.004, -0.0026, 0.0052, 0.0182, -0.0235, -0.0649, -0.015, -0.0475, 0.0184, 0.0576, 0.0477, 0.0064, -0.0041, 0.009, -0.0317, -0.0346, 0.0255, 0.0099, 0.0221, 0.0302, -0.0221, -0.0147, -0.0013, 0.0215, -0.0277, -0.0062, -0.0052, -0.0257, 0.009, 0.0203, 0.0239, 0.0372, -0.0412, -0.031, -0.0686, 0.0043, -0.0354, -0.0034, -0.0148, -0.0483, 0.0007, 0.028, -0.0073, 0.0414, 0.0178, -0.0536, 0.009, -0.0419, 0.0547, 0.0207, -0.0321, 0.0631, -0.0181, -0.096, 0.036, -0.0127, 0.0217, 0.0553, -0.0344, 0.0167, -0.0146]))\n",
      "Row(result=DenseVector([0.0045, -0.0264, 0.0201, -0.0706, -0.001, -0.0639, 0.0241, -0.024, 0.0379, 0.0006, -0.0531, 0.0311, 0.0823, 0.0048, 0.0386, 0.0546, 0.0111, 0.0102, 0.0033, 0.0353, -0.0033, -0.0165, 0.041, -0.0048, 0.0373, -0.0022, 0.041, -0.0125, -0.0108, -0.0023, -0.0187, -0.012, 0.0375, -0.0182, -0.0108, -0.029, 0.0071, 0.05, 0.023, -0.0024, 0.006, 0.0276, 0.0087, -0.0046, -0.0685, -0.022, -0.0565, 0.0157, 0.0515, 0.0457, 0.0008, 0.0113, 0.0155, -0.0368, -0.0432, 0.0111, -0.0005, 0.0291, 0.0487, -0.0095, 0.0008, -0.0053, 0.0308, -0.0021, -0.0144, -0.0059, -0.0242, 0.0067, 0.0023, 0.0117, 0.0215, -0.0441, -0.0288, -0.0748, -0.0118, -0.0404, -0.005, -0.0307, -0.0396, 0.0316, 0.026, -0.0032, 0.033, 0.0221, -0.042, 0.021, -0.0198, 0.051, 0.036, -0.0064, 0.0566, -0.0132, -0.0871, 0.0137, 0.0062, 0.0226, 0.0554, -0.0339, 0.0164, -0.009]))\n",
      "Row(result=DenseVector([-0.0145, -0.0339, 0.0267, -0.0723, 0.0015, -0.0716, 0.0156, -0.0225, 0.0444, -0.0052, -0.0439, 0.0362, 0.08, 0.0158, 0.0337, 0.0451, -0.0088, -0.0008, -0.0094, 0.0342, -0.0027, -0.0157, 0.049, -0.0116, 0.0321, 0.0035, 0.0406, -0.0199, 0.0048, -0.0013, -0.022, -0.0175, 0.0346, -0.0263, 0.0001, -0.0379, 0.0106, 0.046, 0.0182, 0.0039, -0.0119, 0.0158, 0.0183, -0.024, -0.0644, -0.0307, -0.0635, 0.0197, 0.0758, 0.0433, 0.005, 0.0071, 0.0201, -0.0382, -0.0379, 0.018, 0.0128, 0.0146, 0.0342, -0.0335, 0.0064, -0.0029, 0.0217, -0.0127, -0.0129, -0.0015, -0.027, 0.0062, 0.0132, 0.0271, 0.0341, -0.0359, -0.029, -0.0765, -0.0168, -0.0352, -0.0016, -0.027, -0.0406, 0.0222, 0.0276, -0.0047, 0.0493, 0.0184, -0.0586, 0.0132, -0.0287, 0.0456, 0.0326, -0.016, 0.0551, -0.0149, -0.0967, 0.04, -0.0115, 0.0188, 0.0668, -0.032, 0.0224, -0.0098]))\n"
     ]
    }
   ],
   "source": [
    "    #5.将tokenizer得到的分词结果转换数字向量\n",
    "    word2Vec = Word2Vec(vectorSize=100, minCount=0, inputCol=\"words\", outputCol=\"result\")\n",
    "    w2v_model = word2Vec.fit(cv_result)\n",
    "    result = w2v_model.transform(cv_result)\n",
    "    for feature in result.select(\"result\").take(3):\n",
    "        print(feature)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-----+--------------+------------------+-----------------+-------------------------+-----------------------------------+-------------------------+--------------------+--------+----------+-------------------------------------+-------------------+-----+--------------------------------+------------------------------+--------------------+--------------------+\n",
      "|   id|        ztitle|      release_time|       mv_country|              mv_director|                            mv_star|          mv_scriptwriter|            mv_image|mv_score|mv_numbers|                      mv_introduction|            mv_type|intro|                    introduction|                         words|            features|              result|\n",
      "+-----+--------------+------------------+-----------------+-------------------------+-----------------------------------+-------------------------+--------------------+--------+----------+-------------------------------------+-------------------+-----+--------------------------------+------------------------------+--------------------+--------------------+\n",
      "|   49|      大闹天宫|2017-10-13大陆上映|         中国大陆|                     速达|                   李扬\\陈道明\\姚晨|                     速达|https://p1.meitua...|     8.3|      1362|在遥远的东方傲来国，一枚承受了日精...|               动画|     | 在 遥远 的 东方 傲来国 ， 一...|  [在, 遥远, 的, 东方, 傲来...|(5,[0,1,2,3,4],[1...|[-0.0039798310122...|\n",
      "|  123|          龙猫|2018-12-14大陆上映|             日本|                   宫崎骏|             秦岚\\糸井重里\\岛本须美|                   宫崎骏|https://p0.meitua...|     9.1|    11.2万| 小月（日高法子 配音）的母亲（岛本...|动画,冒险,奇幻,家庭|     |  小月 （ 日高 法子   配音 ）...|   [小月, （, 日高, 法子, ,...|(5,[0,1,2,3,4],[1...|[0.00449254804970...|\n",
      "|  784|        甜蜜蜜|2015-02-13大陆上映|         中国香港|                   陈可辛|                 黎明\\张曼玉\\曾志伟|                   陈可辛|https://p1.meitua...|     9.2|     12562| 改革开放初期，黎小军（黎明 饰）告...|          剧情,爱情|     |改革开放 初期 ， 黎小军 （ 黎...|[改革开放, 初期, ，, 黎小军...|(5,[0,1,2,3,4],[1...|[-0.0144980316261...|\n",
      "|  797|兔子镇的火狐狸|2015-10-30大陆上映|         中国大陆|                   葛水英|                 乔诗语\\宣晓鸣\\陈浩|                   葛水英|https://p0.meitua...|     8.2|      2408|兔子镇和狐狸镇是几百年的世代宿敌。...|     动画,冒险,奇幻|     | 兔子 镇 和 狐狸 镇 是 几百年...|   [兔子, 镇, 和, 狐狸, 镇,...|(5,[0,1,2,3,4],[9...|[0.00567104933938...|\n",
      "|  905|    一代宗师3D|2015-01-08大陆上映|中国大陆,中国香港|                   王家卫|                 梁朝伟\\章子怡\\张震|                   王家卫|https://p0.meitua...|     8.2|     39277| 广东佛山人叶问（梁朝伟 饰），年少...|     剧情,动作,传记|     | 广东 佛山人 叶问 （ 梁朝伟  ...|  [广东, 佛山人, 叶问, （, ...|(5,[0,1,2,3],[15....|[-9.4858054433597...|\n",
      "| 1200|        功夫3D|2015-01-15大陆上映|中国大陆,中国香港|                   周星驰|                 周星驰\\黄圣依\\元华|                   周星驰|https://p0.meitua...|     9.3|     26758|    1940年代的上海，自小受尽欺辱的...|     喜剧,动作,犯罪|     |    1940 年代 的 上海 ， 自小...|     [1940, 年代, 的, 上海,...|(5,[0,1,2,3],[10....|[-0.0116279558559...|\n",
      "| 1247|          教父|2015-04-18大陆上映|             美国|     弗朗西斯·福特·科波拉|马龙·白兰度\\阿尔·帕西诺\\詹姆斯·肯恩|     弗朗西斯·福特·科波拉|https://p0.meitua...|     9.3|      1193|      40年代的美国，“教父”维托·唐·...|     犯罪,剧情,惊悚|     |    40 年代 的 美国 ， “ 教父...|    [40, 年代, 的, 美国, ，...|(5,[0,1,2,3,4],[1...|[-0.0129304420048...|\n",
      "| 1287|爱在日落黄昏时|2015-04-20大陆上映|             美国|          理查德·林克莱特|伊桑·霍克\\朱莉·德尔佩\\弗农·多布切夫|          理查德·林克莱特|https://p1.meitua...|     8.7|       195|影片讲述了九年过后一段缘分的再续。...|          剧情,爱情|     | 影片 讲述 了 九年 过后 一段 ...|  [影片, 讲述, 了, 九年, 过...|(5,[0,1,2,3,4],[1...|[-0.0080058343938...|\n",
      "| 1332|      美国往事|2015-04-23大陆上映|      意大利,美国|            赛尔乔·莱翁内|  罗伯特·德尼罗\\詹姆斯·伍兹\\伊丽...|            赛尔乔·莱翁内|https://p1.meitua...|     9.1|       363|  二十年代的美国，纽约少年“面条”（...|          犯罪,剧情|     |二十年代 的 美国 ， 纽约 少年...|  [二十年代, 的, 美国, ，, ...|(5,[0,1,2,3,4],[1...|[0.00477567550791...|\n",
      "| 2972|         教父3|2015-04-20大陆上映|             美国|     弗朗西斯·福特·科波拉|  阿尔·帕西诺\\安迪·加西亚\\索菲亚...|     弗朗西斯·福特·科波拉|https://p0.meitua...|     8.6|       224|    转眼间已经是1979年，第二代教父...|     犯罪,剧情,惊悚|     |    转眼间 已经 是 1979 年 ，...|    [转眼间, 已经, 是, 1979...|(5,[0,1,2,3,4],[1...|[-0.0022443981892...|\n",
      "| 4587|爱在黎明破晓前|2015-04-20大陆上映| 美国,奥地利,瑞士|          理查德·林克莱特|  伊桑·霍克\\朱莉·德尔佩\\安德莉亚...|          理查德·林克莱特|https://p1.meitua...|     8.7|       285|影片讲述了一对陌生男女一天之内在异...|          剧情,爱情|     | 影片 讲述 了 一对 陌生 男女 ...|  [影片, 讲述, 了, 一对, 陌...|(5,[0,1,2,3,4],[7...|[-0.0119659721722...|\n",
      "| 4710|      灵性之光|2015-04-18大陆上映|             波兰|        克日什托夫·扎努西|             Jan Skotnicki\\马辛·...|        克日什托夫·扎努西|https://p1.meitua...|     7.8|         1|赞努西打开国际影坛大门的早期代表作...|               剧情|     |赞努西 打开 国际 影坛 大门 的...| [赞努西, 打开, 国际, 影坛,...|(5,[0,1,2,3,4],[5...|[-0.0141532949263...|\n",
      "| 7641|      机票情缘|      2016大陆上映|             美国|                  唐·罗斯|  本·阿弗莱克\\格温妮斯·帕特洛\\娜...|                  唐·罗斯|https://p0.meitua...|     6.2|         3| 一切本来都完美无瑕，巴迪（本·阿弗...|          剧情,爱情|     |一切 本来 都 完美无瑕 ， 巴迪...| [一切, 本来, 都, 完美无瑕,...|(5,[0,1,2,3,4],[1...|[-0.0375881377080...|\n",
      "| 9839|    石榴的颜色|2015-06-14大陆上映|             苏联|        谢尔盖·帕拉杰诺夫|       索菲柯·齐阿乌列里\\Giorgi ...|        谢尔盖·帕拉杰诺夫|https://p1.meitua...|     8.2|         1|   影片部分根据18世纪亚美尼亚诗人S...|     传记,剧情,音乐|     |  影片 部分 根据 18 世纪 亚美...|    [影片, 部分, 根据, 18, ...|(5,[0,1,2,3,4],[5...|[-0.0265603927173...|\n",
      "| 9927|      杀人短片|2015-04-20大陆上映|             波兰|克日什托夫·基耶斯洛夫斯基| 米罗斯洛·巴卡\\克日什托夫·格洛比...|克日什托夫·基耶斯洛夫斯基|https://p0.meitua...|     8.3|        15|            Jacek Lazar（米罗斯洛·...|          犯罪,剧情|     |          Jacek   Lazar （ 米...|          [jacek, , , lazar...|(5,[0,1,2,3,4],[2...|[-0.0323154084861...|\n",
      "|10757|          缘份|2016-03-25大陆上映|         中国香港|                   黄泰来|               张国荣\\张曼玉\\梅艳芳|                   黄泰来|https://p1.meitua...|     8.6|     28862|在新开通的地下铁中，青年陈保罗（张...|     剧情,喜剧,爱情|     | 在 新开通 的 地下铁 中 ， 青...|  [在, 新开通, 的, 地下铁, ...|(5,[0,1,2,3,4],[1...|[-0.0016196842098...|\n",
      "|11237|      阿飞正传|2018-06-25大陆上映|         中国香港|                   王家卫|               张国荣\\张曼玉\\刘德华|                   王家卫|https://p0.meitua...|     8.8|     21966|英俊不凡的旭仔阿飞是上海移民，他从...|     剧情,爱情,犯罪|     |  英俊 不凡 的 旭 仔 阿飞 是 ...|   [英俊, 不凡, 的, 旭, 仔,...|(5,[0,1,2,3,4],[2...|[-0.0145463338917...|\n",
      "|11792|          侠女|2017-11-03大陆上映|中国台湾,中国香港|                   胡金铨|                     徐枫\\石隽\\白鹰|                   胡金铨|https://p1.meitua...|     8.1|        49| 没落书生顾省斋（石隽 饰）在小镇经...|剧情,动作,武侠,古装|     |  没落 书生 顾省 斋 （ 石隽  ...|  [没落, 书生, 顾省, 斋, （...|(5,[0,1,2,3,4],[1...|[-0.0198847207203...|\n",
      "|11904|        铁道员|2015-06-14大陆上映|             日本|                 降旗康男|             高仓健\\大竹忍\\广末凉子|                 降旗康男|https://p1.meitua...|     8.1|        13|幌舞作为煤矿产地曾繁华过，无奈现今...|          剧情,家庭|     | 幌舞 作为 煤矿 产地 曾 繁华 ...|  [幌舞, 作为, 煤矿, 产地, ...|(5,[0,1,2,3],[16....|[8.91337309145729...|\n",
      "|13457|        龙凤斗|2018-02-10大陆上映|         中国香港|                   杜琪峰|               刘德华\\郑秀文\\吴嘉龙|                   杜琪峰|https://p1.meitua...|     7.9|      1301|  盗生（刘德华 饰）与盗太（郑秀文 ...|     剧情,喜剧,爱情|     |   盗生 （ 刘德华   饰 ） 与 ...|    [盗生, （, 刘德华, , , ...|(5,[0,1,2,3,4],[9...|[-0.0181163564131...|\n",
      "+-----+--------------+------------------+-----------------+-------------------------+-----------------------------------+-------------------------+--------------------+--------+----------+-------------------------------------+-------------------+-----+--------------------------------+------------------------------+--------------------+--------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#6.输出最终结果\n",
    "result.show(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TF-IDF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "StructType(List(StructField(id,IntegerType,true),StructField(ztitle,StringType,true),StructField(release_time,StringType,true),StructField(mv_country,StringType,true),StructField(mv_director,StringType,true),StructField(mv_star,StringType,true),StructField(mv_scriptwriter,StringType,true),StructField(mv_image,StringType,true),StructField(mv_score,StringType,true),StructField(mv_numbers,StringType,true),StructField(mv_introduction,StringType,true),StructField(mv_type,StringType,true),StructField(intro,StringType,true),StructField(introduction,StringType,true),StructField(words,ArrayType(StringType,true),true)))"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from pyspark.ml.feature import HashingTF, IDF, Tokenizer\n",
    "t_words.schema"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Row(features=SparseVector(100, {0: 0.5191, 5: 0.6861, 7: 0.6218, 10: 1.0807, 13: 0.5578, 14: 0.7045, 15: 0.3877, 19: 0.4873, 20: 1.3408, 23: 0.3484, 24: 1.5699, 25: 0.8705, 29: 0.5031, 31: 1.1905, 32: 1.1065, 33: 0.5037, 34: 0.3858, 35: 1.0693, 38: 0.5256, 39: 2.2457, 40: 0.6288, 42: 0.7062, 45: 0.6631, 46: 0.2228, 47: 1.0793, 48: 1.0722, 49: 0.8171, 50: 0.1792, 51: 2.2544, 52: 0.4825, 53: 1.8352, 55: 0.7207, 56: 3.08, 57: 1.6686, 58: 0.8295, 64: 1.6654, 67: 1.4725, 68: 0.3227, 69: 0.8077, 70: 1.5256, 72: 2.1285, 73: 1.4866, 76: 1.6323, 77: 0.8938, 78: 0.7691, 79: 4.2743, 80: 1.1837, 81: 0.5483, 82: 0.5911, 84: 0.3455, 88: 1.3425, 90: 0.9123, 91: 0.6086, 92: 1.7787, 93: 4.5822, 94: 0.6599, 95: 0.8247, 98: 1.8163}), id=49)\n",
      "Row(features=SparseVector(100, {3: 1.9475, 4: 0.668, 5: 0.6861, 6: 0.7198, 7: 1.2436, 9: 0.5044, 10: 0.5404, 12: 3.5278, 13: 0.3718, 14: 1.409, 16: 0.4552, 17: 0.5254, 19: 0.9746, 21: 0.632, 22: 1.4848, 24: 0.5233, 25: 3.4822, 28: 0.6033, 29: 0.5031, 31: 1.4881, 33: 1.5112, 35: 1.0693, 36: 1.0297, 37: 0.6018, 38: 0.2628, 39: 2.2457, 42: 2.1186, 45: 5.3048, 46: 0.8913, 47: 1.619, 50: 0.3136, 51: 2.2544, 52: 1.9301, 54: 0.6936, 55: 1.4414, 57: 2.5029, 60: 0.5995, 62: 1.3294, 63: 0.611, 64: 0.2776, 65: 3.9529, 66: 0.6265, 67: 2.2088, 68: 0.355, 69: 1.6154, 70: 0.7628, 72: 1.935, 75: 1.2797, 77: 1.7875, 79: 1.0686, 80: 1.4797, 81: 0.5483, 82: 0.5911, 83: 1.2623, 84: 0.4398, 87: 0.9473, 88: 0.6712, 90: 0.9123, 92: 0.4447, 93: 0.7637, 94: 0.6599, 95: 0.8247, 96: 0.6753, 97: 0.6542, 98: 0.9082, 99: 0.4631}), id=123)\n",
      "Row(features=SparseVector(100, {0: 0.5191, 2: 1.7978, 3: 1.558, 4: 0.668, 5: 0.6861, 7: 1.2436, 10: 1.0807, 11: 1.4107, 12: 1.1759, 13: 0.1859, 15: 1.163, 17: 0.5254, 18: 0.753, 21: 1.2639, 24: 0.5233, 25: 1.7411, 31: 1.1905, 32: 0.7377, 33: 0.5037, 34: 1.5432, 36: 0.5149, 37: 0.6018, 39: 0.7486, 42: 0.7062, 44: 2.9039, 46: 0.4457, 47: 1.0793, 48: 1.0722, 50: 0.1792, 51: 1.6908, 52: 0.4825, 54: 1.3871, 55: 0.7207, 56: 0.77, 61: 0.7319, 62: 0.6647, 64: 1.1103, 65: 0.4941, 67: 0.7363, 68: 0.355, 69: 0.8077, 70: 0.7628, 72: 1.7415, 74: 4.2344, 77: 0.8938, 80: 1.7756, 81: 1.0966, 82: 1.1823, 84: 0.2827, 86: 5.7814, 89: 0.9366, 90: 0.9123, 92: 0.8893, 93: 1.5274, 96: 0.6753, 97: 1.3084, 99: 1.3893}), id=784)\n",
      "Row(features=SparseVector(100, {0: 0.5191, 2: 8.9889, 3: 0.3895, 6: 0.7198, 12: 0.392, 13: 0.5578, 14: 1.409, 15: 1.9383, 17: 1.0508, 18: 1.506, 20: 0.6704, 21: 0.632, 22: 0.7424, 23: 1.3936, 24: 1.0466, 26: 1.3116, 31: 0.5952, 32: 0.5532, 33: 1.5112, 34: 0.7716, 36: 0.5149, 37: 1.2035, 38: 1.0511, 42: 0.7062, 45: 1.3262, 46: 0.2228, 49: 1.6342, 50: 0.3584, 51: 0.5636, 52: 0.9651, 54: 1.3871, 57: 0.8343, 58: 0.8295, 59: 1.7272, 60: 0.5995, 62: 1.9942, 63: 0.611, 64: 0.8327, 66: 3.1323, 68: 0.2905, 69: 0.8077, 71: 1.5078, 72: 1.3545, 73: 0.7433, 75: 1.2797, 76: 0.8162, 78: 0.7691, 80: 0.8878, 82: 1.7734, 83: 1.8935, 84: 0.2199, 85: 0.791, 88: 1.3425, 89: 0.9366, 91: 0.6086, 93: 0.7637, 94: 1.9796, 95: 0.8247, 97: 7.1964, 98: 0.9082, 99: 0.9262}), id=797)\n",
      "Row(features=SparseVector(100, {2: 2.6967, 3: 0.779, 4: 0.668, 5: 1.3721, 8: 0.6179, 9: 1.5133, 10: 1.6211, 11: 2.116, 13: 0.5578, 14: 2.818, 16: 0.4552, 17: 1.5763, 21: 1.2639, 23: 0.6968, 25: 0.8705, 27: 1.8498, 28: 0.6033, 29: 1.5092, 30: 0.7459, 31: 2.0833, 32: 0.3688, 33: 1.5112, 34: 0.7716, 38: 0.2628, 39: 0.7486, 41: 0.7053, 42: 0.7062, 46: 0.4457, 47: 1.0793, 49: 0.8171, 50: 0.1792, 51: 1.1272, 52: 1.4476, 55: 0.7207, 56: 2.31, 58: 4.9769, 59: 1.7272, 60: 1.1989, 61: 1.4638, 62: 1.3294, 63: 1.2219, 64: 3.0532, 65: 0.9882, 66: 1.8794, 67: 2.9451, 68: 0.4841, 69: 0.8077, 70: 0.7628, 71: 0.7539, 72: 2.9025, 75: 1.2797, 76: 0.8162, 77: 1.7875, 78: 0.7691, 79: 1.0686, 80: 2.0715, 82: 1.1823, 84: 0.2199, 85: 1.5819, 86: 1.9271, 87: 0.9473, 88: 2.685, 89: 0.9366, 90: 0.9123, 92: 3.1127, 93: 1.5274, 99: 0.9262}), id=905)\n",
      "Row(features=SparseVector(100, {0: 1.0382, 4: 1.336, 6: 0.7198, 9: 1.0089, 11: 0.7053, 12: 0.392, 13: 1.3015, 14: 0.7045, 15: 0.3877, 17: 2.1017, 18: 1.506, 20: 0.6704, 21: 1.2639, 22: 2.9696, 23: 0.6968, 25: 2.6116, 26: 0.6558, 27: 0.9249, 28: 0.6033, 29: 0.5031, 30: 1.4918, 31: 0.2976, 32: 0.5532, 34: 1.1574, 36: 1.0297, 37: 1.2035, 38: 0.5256, 39: 1.4971, 40: 0.6288, 42: 0.7062, 43: 0.6828, 44: 3.8718, 46: 2.0055, 48: 1.0722, 49: 0.8171, 50: 0.224, 51: 0.5636, 52: 0.9651, 61: 1.4638, 63: 1.2219, 64: 0.2776, 66: 1.2529, 67: 1.4725, 68: 0.3227, 70: 0.7628, 72: 0.387, 75: 1.2797, 76: 0.8162, 77: 0.8938, 78: 1.5382, 80: 1.4797, 84: 0.1571, 88: 0.6712, 89: 0.9366, 91: 0.6086, 92: 1.7787, 96: 2.7014, 99: 0.4631}), id=1200)\n",
      "Row(features=SparseVector(100, {0: 0.5191, 1: 0.8948, 2: 1.7978, 3: 1.1685, 4: 1.336, 5: 0.6861, 6: 2.1595, 9: 0.5044, 10: 1.0807, 11: 4.232, 13: 1.4874, 14: 1.409, 15: 2.7136, 18: 1.506, 19: 1.4619, 23: 0.6968, 24: 1.0466, 26: 0.6558, 27: 1.8498, 28: 3.0164, 31: 0.8929, 32: 0.1844, 33: 1.0075, 34: 0.3858, 35: 3.7425, 36: 0.5149, 38: 1.0511, 39: 0.7486, 40: 0.6288, 41: 0.7053, 42: 2.8247, 44: 6.7757, 46: 1.5598, 49: 0.8171, 50: 0.336, 51: 0.5636, 52: 1.9301, 54: 0.6936, 59: 1.7272, 60: 0.5995, 62: 1.9942, 64: 0.8327, 65: 0.9882, 68: 0.4196, 71: 0.7539, 72: 1.7415, 74: 0.8469, 79: 1.0686, 80: 0.5919, 81: 0.5483, 83: 0.6312, 84: 0.3455, 85: 0.791, 87: 2.8418, 88: 0.6712, 89: 0.9366, 92: 0.8893, 93: 0.7637, 94: 0.6599, 95: 2.4741, 96: 2.026, 97: 0.6542, 98: 1.8163, 99: 1.8524}), id=1247)\n",
      "Row(features=SparseVector(100, {0: 0.5191, 1: 0.8948, 5: 0.6861, 7: 0.6218, 9: 0.5044, 12: 0.784, 13: 0.3718, 14: 1.409, 16: 0.4552, 17: 1.0508, 18: 2.259, 20: 0.6704, 21: 1.2639, 23: 1.0452, 24: 0.5233, 27: 0.9249, 28: 2.4131, 31: 0.5952, 32: 2.5818, 34: 1.1574, 35: 1.0693, 36: 0.5149, 39: 1.4971, 40: 0.6288, 43: 1.3655, 48: 1.0722, 50: 0.1792, 51: 0.5636, 52: 0.4825, 53: 0.6117, 54: 0.6936, 55: 0.7207, 56: 0.77, 59: 3.4544, 60: 0.5995, 61: 0.7319, 62: 0.6647, 64: 1.6654, 65: 0.4941, 66: 1.2529, 68: 0.355, 72: 0.9675, 74: 0.8469, 75: 1.2797, 76: 1.6323, 77: 1.7875, 78: 1.5382, 80: 0.5919, 81: 1.0966, 84: 0.2513, 86: 0.9636, 87: 2.8418, 89: 0.9366, 91: 1.8259, 92: 0.4447, 93: 0.7637, 94: 3.9592, 97: 1.3084}), id=1287)\n",
      "Row(features=SparseVector(100, {0: 0.5191, 3: 0.779, 4: 1.336, 5: 0.6861, 6: 0.7198, 7: 0.6218, 10: 2.7019, 12: 0.392, 13: 3.1607, 14: 1.409, 15: 1.163, 16: 0.4552, 17: 0.5254, 18: 1.506, 19: 0.4873, 20: 0.6704, 21: 0.632, 22: 0.7424, 23: 1.3936, 25: 1.7411, 26: 1.3116, 28: 1.8099, 31: 1.1905, 32: 0.5532, 33: 1.0075, 34: 0.3858, 35: 1.0693, 36: 1.5446, 37: 1.2035, 39: 2.2457, 41: 2.116, 42: 0.7062, 43: 0.6828, 45: 0.6631, 46: 2.8968, 47: 0.5397, 48: 3.2167, 49: 0.8171, 50: 0.4703, 51: 3.3816, 52: 0.4825, 53: 0.6117, 55: 1.4414, 56: 0.77, 58: 0.8295, 60: 0.5995, 61: 0.7319, 62: 1.9942, 63: 0.611, 64: 1.1103, 65: 0.4941, 66: 1.2529, 67: 0.7363, 68: 0.6455, 71: 2.2616, 72: 1.161, 74: 0.8469, 75: 2.5594, 79: 1.0686, 80: 1.4797, 81: 1.6449, 82: 0.5911, 84: 0.5654, 87: 0.9473, 89: 7.4924, 92: 0.8893, 93: 1.5274, 95: 2.4741, 98: 0.9082, 99: 0.4631}), id=1332)\n",
      "Row(features=SparseVector(100, {0: 0.5191, 2: 0.8989, 3: 0.3895, 4: 0.668, 5: 2.0582, 6: 0.7198, 9: 1.0089, 13: 0.1859, 14: 1.409, 15: 1.163, 18: 0.753, 19: 0.9746, 21: 1.2639, 22: 0.7424, 23: 1.0452, 24: 1.0466, 25: 0.8705, 26: 3.2791, 28: 1.2066, 29: 1.0061, 30: 0.7459, 31: 1.1905, 32: 0.3688, 33: 1.0075, 34: 0.3858, 35: 3.7425, 38: 1.0511, 41: 1.4107, 42: 0.7062, 43: 0.6828, 44: 2.9039, 45: 1.3262, 46: 0.4457, 47: 0.5397, 49: 0.8171, 50: 0.2464, 52: 0.9651, 54: 0.6936, 55: 0.7207, 57: 0.8343, 58: 1.659, 59: 6.0451, 61: 0.7319, 64: 0.8327, 65: 0.4941, 66: 0.6265, 67: 1.4725, 68: 0.5487, 70: 0.7628, 71: 1.5078, 72: 1.3545, 73: 0.7433, 75: 0.6399, 77: 1.7875, 78: 0.7691, 79: 1.0686, 80: 1.4797, 83: 0.6312, 84: 0.2513, 87: 0.9473, 88: 0.6712, 91: 0.6086, 93: 0.7637, 94: 0.6599, 96: 0.6753, 97: 0.6542, 98: 0.9082, 99: 1.8524}), id=2972)\n"
     ]
    }
   ],
   "source": [
    "hashingTF = HashingTF(inputCol=\"words\", outputCol=\"rawFeatures\", numFeatures=100)\n",
    "featurizedData = hashingTF.transform(t_words)\n",
    "# alternatively, CountVectorizer can also be used to get term frequency vectors\n",
    "\n",
    "idf = IDF(inputCol=\"rawFeatures\", outputCol=\"features\")\n",
    "idfModel = idf.fit(featurizedData)\n",
    "rescaledData = idfModel.transform(featurizedData)\n",
    "for features_label in rescaledData.select(\"features\", 'id').take(10):\n",
    "    print(features_label)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.ml.feature import Normalizer\n",
    "normalizer = Normalizer(inputCol=\"features\", outputCol=\"norm\", p=2.0)\n",
    "df_norm = normalizer.transform(rescaledData)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pyspark.sql.functions as psf\n",
    "from pyspark.sql.types import DoubleType\n",
    "dot_udf = psf.udf(lambda x,y: float(x.dot(y)), DoubleType())\n",
    "similarity_idf = df_norm.alias(\"i\").join(df_norm.alias(\"j\"), psf.col(\"i.ID\") < psf.col(\"j.ID\"))\\\n",
    "    .select(\n",
    "        psf.col(\"i.ID\").alias(\"i\"), \n",
    "        psf.col(\"j.ID\").alias(\"j\"), \n",
    "        dot_udf(\"i.norm\", \"j.norm\").alias(\"dot\"))\\\n",
    "    .sort(\"i\", \"j\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+---+---+-------------------+\n",
      "|  i|  j|                dot|\n",
      "+---+---+-------------------+\n",
      "| 49|123|0.47207839238120425|\n",
      "| 49|784|0.36394728727439885|\n",
      "+---+---+-------------------+\n",
      "only showing top 2 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "similarity_idf.show(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 创建连接写入数据\n",
    "similarity_idf.write.format(\"jdbc\").option(\"url\", \"jdbc:mysql://localhost:3308/meich_db\")\\\n",
    ".option(\"dbtable\", \"sim_idf\").option(\"user\", \"root\").option(\"password\", \"Mysql_08\").mode('append').save()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 利用2范数求余弦相似度"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.ml.feature import Normalizer\n",
    "normalizer = Normalizer(inputCol=\"result\", outputCol=\"norm\", p=2.0)\n",
    "data = normalizer.transform(result)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pyspark.sql.functions as psf\n",
    "from pyspark.sql.types import DoubleType\n",
    "dot_udf = psf.udf(lambda x,y: float(x.dot(y)), DoubleType())\n",
    "similarity_w2v = data.alias(\"i\").join(data.alias(\"j\"), psf.col(\"i.ID\") < psf.col(\"j.ID\"))\\\n",
    "    .select(\n",
    "        psf.col(\"i.ID\").alias(\"i\"), \n",
    "        psf.col(\"j.ID\").alias(\"j\"), \n",
    "        dot_udf(\"i.norm\", \"j.norm\").alias(\"dot\"))\\\n",
    "    .sort(\"i\", \"j\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+---+---+------------------+\n",
      "|  i|  j|               dot|\n",
      "+---+---+------------------+\n",
      "| 49|123|0.9332584371505092|\n",
      "| 49|784|0.9642866951047674|\n",
      "+---+---+------------------+\n",
      "only showing top 2 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "similarity_w2v.show(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 创建连接写入数据\n",
    "similarity.write.format(\"jdbc\").option(\"url\", \"jdbc:mysql://localhost:3308/meich_db\")\\\n",
    ".option(\"dbtable\", \"sim\").option(\"user\", \"root\").option(\"password\", \"Mysql_08\").mode('append').save()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.7.2",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
